DOCUMENT RESUME 



ED 457 352 



CE 082 368 



AUTHOR 

TITLE 

PUB DATE 
NOTE 

AVAILABLE FROM 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Herod, L. 

Integrating Technology into Canadian Adult Literacy 
Programs: Curriculum Evaluation Considerations. 
2000-04-00 
14p . 

For full text: 

http : //www.nald. ca/f ull text /herod/april /cover . htm. 
Information Analyses (070) 

MF01/PC01 Plus Postage. 

* Adult Literacy; Case Studies; Comparative Analysis; 
*Computer Uses in Education; *Curriculum Evaluation; 
*Educational Technology; * Evaluation Methods; Foreign 
Countries; *Literacy Education; Models; Participatory 
Research; Technological Advancement; Technology Transfer 
*Canada 



ABSTRACT 



The integration of technology into adult literacy promises . 
both exciting and problematic times, especially with regard to curriculum 
evaluation. Different evaluation approaches have various strengths and 
weaknesses. A particular strength of the adversarial approach is a 
well-rounded evaluation with both positives and negatives identified; its 
main weakness is that it is less inclusive. The main strength and weakness of 
the comparative and case study approaches is that their best use is to 
evaluate standardized curriculum. The most effective is the participatory 
approach. Its main benefit is its capture of the program context; however, it 
depends on the ability of many stakeholders to reach an accommodation of 
potentially divergent interests and opinions. Three classic evaluation models 
have their own strengths and weaknesses. The discrepancy evaluation model 
provides a great deal of information regarding a program at all stages, but 
it is resource- heavy and does not capture the qualitative aspects. The CIPP 
(context- input-process-product) model's main strength and weakness is its 
ongoing and comprehensive nature. One main strength of the 

contingency- congruency model that could also be considered a weakness is its 
deliberate search for judgements regarding the value of various aspects from 
a wide range of stakeholders. Aspects of technology must be considered in any 
curriculum evaluation: different types, computer use, and desirability and 
impact of technology. (Contains 38 references.) (YLB) 
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INTEGRATING TECHNOLOGY INTO CANADIAN ADULT LITERACY PROGRAMS: 
CURRICULUM EVALUATION CONSIDERATIONS 

L. Herod, MEd, BA, April 2000 



INTRODUCTION 

The integration of technology-computers in particular-into curricula is a topical issue 
among educators in general. Among adult literacy practitioners it is particular so. 
Although the incorporation of computers into adult literacy teaching and learning is 
heralded by most (Amos, 1998; Ginsburg, 1999; Hopey, 1998; Hopey, Harvey-Morgan 
& Rethemeyer, 1996; Sabatini & Ginsburg, 1999; Stites, 1999; Stites, Hopey & 
Ginsburg, 1998; Turner, 1999; Wagner & Hopey, 1998; Wilson, 1998), there are definite 
concerns. These include: 

• the philosophical (i.e., the place of technology within literacy teaching and 
learning); 

• the practical (i.e., the use of very scarce resources-money, time and 
personnel-to support technology); and, 

• the educational (i.e., the effectiveness of technology with regard to literacy 
teaching and learning). 

This paper explores the concept of curriculum evaluation framed against the issue of 
integrating technology into adult literacy programs. While evaluation is crucial to 
providing effective and relevant curricula at any time, it is crucial at a point in time when 
many believe the pressure of technology will undoubtedly result in changes to the field, 
"We all acknowledge that the information age has had a profound impact on the world 
around us; thus it is not unreasonable to posit that the information age should also 
affect the form and function of adult education" (Ginsburg, 1999, p. 45). 



CLARIFYING TERMS 

Many educators have rightly noted that there is a lack of clarity in curriculum-related 
terminology (Lewy, 1977; Madaus & Kellaghan, 1992; Wolf, 1990). Miller & Seller 
(1990), for example, suggest that the meaning of "curriculum" varies widely among 
practitioners: 

At one end, curriculum is seen merely as a course of study; at the other end, 
curriculum is more broadly defined as everything that occurs under the auspices 
of the [organization]. In the middle of the spectrum, curriculum is viewed as an 
interaction between students and teachers that is designed to achieve specific 
educational goals (p. 3). 

As such, it is prudent to define several terms. For the purposes of this paper, the term 
"curriculum" shall be used in the broadest possible sense and will include content, 
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resources, process (teaching approach and learning strategies), learners, practitioners, 
and what Schwab (1993) terms "milieus." "Milieus" refer to political, economic, 
community/social, and personal factors such as religion, ethnic background, which 
influence a curricular environment. Madaus & Kellaghan (1992) similarly note that "the 
terms evaluation and assessment do not have generally agreed-upon meanings." Often 
they are used interchangeably when in fact "assessment" refers most commonly to the 
appraisal of individual learners in terms of their knowledge and/or skills. "Evaluation," on 
the other hand, has a broader application to curriculum proper and includes those 
aspects noted under the definition of "curriculum" above. 

THE FUNCTIONS OF EVALUATION 

As Wolf (1990) aptly suggests, "Educational evaluation is clearly decision-oriented. It is 
intended to lead to better policies and practices in education." Beyond this, however, 
discussions are commonly framed against the related notions of formative and 
summative purposes (Eisner, 1979; Lewy, 1977; Wolf, 1990; Worthen, 1990), terms first 
offered by Scriven in 1967. Simply put, "formative" evaluation is ongoing and takes 
place before and during the development of a particular program, while "summative" 
evaluation takes place upon its completion. In addition to providing information about 
what to modify within a program (e.g., curricular materials, organization, teaching 
approach), formative evaluation requires stakeholders to make judgements regarding 
the worth or value of a program; that is, regarding implementation. As Ornstein & 
Hunkins (1998) suggest, "In determining the value of a curriculum plan, educators must 
eventually ask whether the results they expect to obtain are worth what the cost of 
delivering them is likely to be" (p. 323). This phase of evaluation is crucial for adult 
literacy programs since there are many philosophical, practical and educational 
concerns among practitioners regarding the "cost" and relevance of integrating 
technology. Front-end evaluation would allow adult literacy practitioners to clarify, make 
decisions, and act on this issue rather than fall victim to the 'steamroller' that is 
technology (Chandler, 1995; Postman, 1993; Robertson, 1998). Summative evaluation 
provides information regarding whether or not the objectives of the program were met. 

APPROACHES 

There appears to be a paucity of research into educational evaluation in general. As 
Worthen (1990) writes, "There is still little empirical information about the relative 
efficacy of alternative evaluation plans or techniques or many evaluation components 
germane to almost any model" ( p. 46). At one time, data gathered and interpreted 
according to the scientific method (i.e. qualitative research), was considered the only 
acceptable and legitimate data. More recently, however, educators have begun to 
suggest that qualitative approaches and the descriptive data they provide are as 
relevant as quantitative approaches (Eisner, 1979; Fetterman, 1988; Ornstein & 
Hunkins, 1998; Walker, 1978). Indeed, many propose that both approaches are 
essential to gaining a full understanding of the curriculum being evaluated. Ornstein & 
Hunkins (1998), for example, write: 

Those urging use of humanistic or naturalistic procedures argue for a more 

holistic engagement that presents us with much more detail. With these methods, 
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we get portraits of the situations we are evaluating. Evaluation reports are less 
lists of numbers than they are written descriptions of what was found or what 
occurred. The approach focuses more on human interactions than on outcomes 
and more on the quality that the quantity of classroom or school life. Scientific 
approaches may have revealed data that depicted what people did, but 
naturalistic approaches, while interested in that, are also concerned about 
delving into the why behind the what of performance (p. 332). 

This has led to a call for pluralistic approaches to evaluation. The main barrier to using 
pluralistic approaches for most areas in education involves resources such as finances, 
time, and particularly expertise (Borich & Jemelka, 1982; Miller & Seller, 1990). This is 
magnified in adult literacy programs for several reasons: 

• programs are underfunded; 

• programs are understaffed and rely heavily on volunteers, thus full/part-time 
staff have very limited time (much of which is spent fund-raising to keep the 
program going); and 

• curricula are less standardized and formalized than in many other areas of 
education, thus, expertise in curriculum development and evaluation is 
minimal. 

That said, as in other areas of the curriculum development cycle, knowledge of various 
approaches to evaluation and understanding of their applicability to individual evaluation 
situations will maximize the effectiveness of any effort. Several different approaches are 
outlined in the following section and their strengths and weaknesses noted. 

Adversarial Approach 

This approach to evaluation is similar to a debate in that it involves evaluators taking up 
opposite positions regarding a curriculum (Ornstein & Hunkins, 1998; Waters, 1998; 
Worthen, 1990). A particular strength of this approach is that it can result in well- 
rounded evaluation in the sense that both positives and negatives are deliberately 
identified and discussed. In other approaches there is the potential at least for the 
evaluation to be skewed, either intentionally or unintentionally. The main weakness of 
this approach is that it is less inclusive than other approaches; that is, it is a somewhat 
competitive process involving teams of evaluators versus a collaborative effort among a 
variety of curriculum stakeholders. As such, although strengths and weaknesses are 
identified, these may somewhat narrow since stakeholder input is neglected. In terms of 
the applicability of this approach to adult literacy, it may be that it would be most useful 
at the national and/or provincial level. That is, it would be useful for practitioners to 
evaluate the philosophical, practical and educational aspects surrounding the issue of 
technology in a global manner before moving onto more detailed, specific evaluation at 
the regional and local level. 

Comparative and Case Study Approaches 

As its name indicates, this approach compares various aspects of curriculum (Eisner, 
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1979). Both its main strength and weakness is that it is best used to evaluate 
standardized curriculum. In view of this, its use in adult literacy would be limited since 
programs are individualized in many respects (e.g., curriculum - informal, non- 
standardized; focus - workplace literacy, family literacy, ABE/GED; group size - small 
group versus one-to-one tutoring). A case study approach (Saez & Carreto, 1998), in 
which the minutiae of individual curriculum are more easily captured and evaluated 
would be the better choice. Although there is much to be gained from both approaches, 
it is unlikely that either will be used by the field anytime in the near future; the 
comparative approach because historically provinces/regions do not conduct 
evaluations, and the case study approach because it requires a degree of expertise 
lacking in the field. 

Participatory Approach 

The most effective approach for use in the field is likely the participatory approach. 
Simply put, participatory evaluation is conducted by program participants for program 
participants, guided by trained evaluation personnel. Participants or "stakeholders" 
(Brandon, 1999) may include practitioners/academics, learners, volunteers, Board 
members, and representatives of what was described earlier as program "milieus" 
(Schwab, 1976) (e.g., community members, politicians, business people) The main 
benefit of this approach is that it captures the context of the program and thus, the 
needs/wants/interests of a wide range of stakeholders, more fully than other 
approaches. From a purely practical point of view, as Miller & Seller (1990) point out, 
"Acknowledgement of different points of view about the effectiveness of a curriculum 
provides more alternatives to the decision makers" (p. 324). Thus, putting many heads 
together will ostensibly result in "painting the bigger picture." And therein lies the 
potential weakness of this approach; the process is dependent on the ability of many 
stakeholders to reach an accommodation of potentially divergent interests and opinions. 
It is why Schwab (1976), for example, strongly recommended the inclusion of a 
curriculum specialist who would facilitate the process. It is quite likely that most adult 
literacy programs would experience success with participatory evaluation since the 
prevailing philosophy in the field can be described as humanistic or "transformational" 
Miller and Seller (1990, p. 8). That is, the emphasis of curriculum is on social and 
personal change (Barer-Stein & Draper, 1991; MacKeracher, 1996; Merriam & 
Caffarella, 1991; Selman & Dampier, 1991). 



MODELS 

Although there are a wide variety of evaluation models, three 'classic' ones are 
described below. Each has its own strengths and weaknesses and these are noted for 
each. Choosing a model will depend on understanding these, as well as the intended 
purpose of the particular model. This is also noted for each. 

Discrepancy Evaluation Model 

This model was proposed by Provus (Miller & Seller, 1990; Ornstein & Hunkins, 1998) 
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and essentially looks for any discrepancies between the standards and performance of 
a program. The information is gathered using a quantitatively-oriented approach and is 
passed to program decision-makers to act upon in each of the following five areas: 

• design - should the program be implemented, modified or rejected; 

• installation - is the program operating as it was intended; 

• processes- are the interactions, teaching approach, etc., adequate; 

• products - did the program achieve the objectives it was supposed to; and, 

• cost - were the benefits worth the costs? 

The main strength of this model is that it provides a great deal of information regarding 
a program at all stages for decision-makers. In terms of adult literacy, however, this 
model has two main weaknesses: a) it is resource-heavy; and, b) it does not capture the 
qualitative aspects of a program. Although the management-related aspects of a 
program are important, in adult literacy they are secondary or at least on par with 
humanistic concerns. For example, in the 1990's when the Harris government was 
brought into power in Ontario, an evaluation of literacy programs was conducted which 
focused on the success of programs in terms of employment-related learning objectives. 
That is, were programs contributing to the employability of participants? While this is a 
goal that many literacy learners are working toward, many simply seek to improve the 
quality of their lives and those around them. They may wish to become less dependent 
on family and friends, they might want to be better able to help their children with 
homework, and so on. The Ontario government evaluation did not capture what might 
be described as the more qualitative aspects of literacy learning. Participants in general 
"transform" (Miller & Seller 1990) into more fully-functioning individuals, family members 
and citizens who may or may not be employed, but who contribute in many equally 
important ways to society nevertheless. 

CIPP (Context-Input-Process-Product) Model 

This model was developed by Stufflebeam (Miller & Seller, 1990; Ornstein & Hunkins, 
1998; Worthen, 1990) and much like the Provus model, several types of evaluation 
information are gathered for decision-makers to act upon. These include: 

• program context - the environment in which the program currently operates, 
including both internal and external factors, and the expected and actual current 
performance of the program; 

• input - examines the feasibility of alternatives for meeting program objectives with 
the most effective use of resources; 

• process - examines the efficacy of the program design from an operational or 
procedural perspective; and, 

• product - examines whether or not anticipated objectives have been achieved. 
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The main strength of the CIPP model is also its main weakness, at least for adult 
literacy programs. That is, it is ongoing and very comprehensive, thus evaluations using 
this model are resource-heavy, far beyond what the majority of programs could 
manage. In addition, much like the Provus model evaluation information in this model is 
gathered for program managers and in a quantitatively oriented manner. While there 
may be instances when this would useful to adult literacy programs, there is a greater 
need for models which adopt a participatory approach in which collaborative decision- 
making among a wider range of stakeholder is allowed for. In addition, models which 
adopt a combination of qualitative and quantitative methods would be more in keeping 
(and thus provide capture valuable information) with the "transformational" orientation of 
this field. 

Contingency-Congruency Model 

This model was developed by Robert Stake (Miller & Seller, 1990; Ornstein & Hunkins, 
1998; Worthen, 1990) and in contrast to the Provus and Stufflebeam models, uses a 
qualitatively oriented and collaborative approach versus a management oriented 
approach in which information is gathered for program managers. Three basic types of 
information are gathered in this model including: 

• antecedents - information regarding factors in the educational environment that 
exist prior to the program and which may affect outcomes such as learner 
characteristics (e.g., prior performance, psychological tests), teacher 
characteristics (e.g., years of experience, subject matter expertise), and 
resources (e.g., availability, quality); 

• transactions - information regarding the process of teaching and learning such as 
teacher-student interactions, interactions with curricular materials, environment 
interactions (e.g., organization of time allotment, space arrangement); and, 

• outcomes - information regarding both short and long-term 'products' of the 
program such as changes in cognition/affect, not only with regard to students, but 
in other stakeholders such as teachers, administrators, and in the greater 
community. 

In each of the three areas or phases of a program, Stake's model examines what was 
intended and what was actually observed. One of the main strengths of this model in 
terms of adult literacy programs is that it deliberately seeks judgements regarding the 
value or worth of various aspects of the phases from a wide range of stakeholders. 
Conversely, this may be considered a weakness in that a great deal of skill and 
experience may be required on the part of the evaluator to accommodate divergent 
views/needs/wants. However, based on the underlying philosophy of adult literacy 
(humanistic, democratic, transformational), this represents a particular strength. As with 
the Stufflebeam and Provus model, however, this approach is resource heavy and as 
such, makes it a dubious choice. 




Evaluation Strategy for Non-Specialists 

As in the case of determining an approach to evaluation, one’s choice of models will 
depend on many factors, not the least of which is the availability of experienced 
evaluators. The lack of curricular expertise in the field of adult literacy severely limits the 
likelihood that any of the above-mentioned models would be used since they require a 
knowledgeable specialist and substantial resources. However, this is not to say that 
evaluation should not be conducted at all. Rather, an awareness of evaluation 
approaches and models, combined with will and a simpler strategy can go a long way 
toward effective evaluation. As cited by Miller and Seller (1990, p. 324-327), Leithwood 
offers just such a strategy: 

• Define the evaluation problem (e.g., define the problem, identify stakeholders, 
generate a list of ranked decisions to be made) 

• Choose an appropriate methodology (e.g., identify the questions to be asked, 
sources of information, methods for collecting data, etc) 

• Plan for data collection, analysis and reporting (e.g., identify the major tasks, 
sub-tasks, and responsibilities, and develop a timeline) 

• Collect and compile data (e.g., select instruments and samples, collect and 
compile data) 

• Analyze and report data (e.g., summarize data, refine and apply decision 
rules, report results) 

This leaves room to use a 'toolbox' approach; that is, to fill in the details at each stage 
with bits and pieces taken from the various approaches/models which, are relevant to 
the particular situation, and suitable to the skill of the evaluator(s) and available 
resources. 

CONSIDERATIONS 

In closing, a review of the literature regarding technology in adult literacy indicates that 
there are many aspects of technology which will need to be considered in any 
curriculum evaluation (Amos, 1998; Cort & Disario, 1996; Garner, Dilloway & Whiten, 
1999; Ginsburg, 1999; Hopey, 1998; Hopey, Harvey-Morgan & Rethemeyer, 1996; 
Rosen, 1997; Sabatini & Ginsburg, 1999; Provenski; 1999; Stites, 1999; Stites, 1998; 
Stites, Hopey & Ginsburg, 1998; Turner, 1999; Wagner & Hopey, 1998; Wilson & Javed, 
1998). These are loosely grouped below and represent a beginning versus exhaustive 
list: 

Different Types of Technology 

Although the main technology to be evaluated would be computers, programs also have 
other types of technology that would also need to be included in any evaluation: These 
might include: 
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• Other Instructional Technologies - technology used in instruction other than 
the computer such as video/audio-conferencing, video/audio cassettes, 
television, etc. 

• Assistive Technologies - various hardware and software (e.g., tape recorders, 
computers, calculators, speech-to-text writers, etc) used to assist students 
with learning disabilities. 

• Administrative Support Technologies - (e.g., telephone, fax, photocopier, etc). 
Computer Use 

• Instructional: 

- The computer as curriculum (i.e., operation of the computer and software) 

- The computer is used to deliver instruction (i.e., curriculum is delivered via 
the computer - online course, a software program, etc) 

- The computer is used to support instruction (e.g., practitioners/tutors use 
various software programs and the Internet to develop curriculum, learners 
use the Internet for research, use various software programs to complete 
learning activities, etc). 

• Administrative (e.g., communicate via email, record keeping, word 
processing, data storage, accounting, fund-raising, disseminate program 
information) 

• Technical (installing, maintaining, fixing and upgrading equipment and 
software) 

Desirability and Impact of Technology 

These considerations can be grouped into the themes described in the introduction to 
this paper: 

• Philosophical 

- Time, effort and knowledge will be required to integrate computers into the 
curriculum-will this overtax programs that are already stretched? 

- Do learners and staff want to learn/use technology? Do they feel that they 
benefit from its use? 

• Practical 

- Are there enough computers and are they in working order? 

- Is the Internet available, reliable and affordable? 

- Are the operating systems, software programs and peripheral equipment 
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such as printers easy to use? 

- Is there a plan in place to purchase, install, maintain and upgrade equipment 
and programs? 

- Is there a professional development plan in place for staff/volunteers to learn 
the computer? 

• Educational Efficacy 

- Are low literacy learners able to learn to use the computer? 

- How can the computer be used to promote and/or enhance learning? What 
features may detract from learning? 

- Are staff able to develop curriculum which integrates applies the computer in 
learning in an effective and relevant manner? Is there an adequate 
professional development plan in place to assist with this? 

CONCLUSION 

The integration of technology into adult literacy promises both exciting and problematic 
times for the field of adult literacy. It is through evaluation efforts at all levels that best 
practices and lessons learned will be identified and the field will move forward in a 
relevant and effective way. One such detailed evaluation effort is Cyberstep, a 
ambitious and comprehensive three-year curriculum development and evaluation 
project out of the Vocational and Adult Education Department of California State 
University which promises to provide a great deal of valuable information regarding 
technology and curriculum standards (i.e., content, assessment, technology, interface, 
management system, and template). One aspect of this project that may be particularly 
appealing for Canadian adult literacy practitioners is the creation of course content and 
assessment "templates" that can be individualized by users to suit their particular 
needs. A similar effort in Canada would greatly speed the creation and lower the cost of 
creating instructional materials. It is, however, crucial that practitioners look to 
evaluation efforts such as these, as well as conduct their own evaluations, in order to 
learn about the best and worst that technology can offer adult literacy: 

The potential for technology to expand and improve learning by adults is 
especially great. To take advantage of technology’s potential, adult educators, 
planners, and policymakers need to critically access the performance of the 
technology and the quality of learning that technology supports (Stites et al, 

1998, p. 1). 
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